N number of servers running simultaneously (to support Y number of agents)

• servers need to be the same version

how does the server expose itself? - the "public route" to the server (single ip/port) that we expect the agents to try the server on will be written to the HA tables

all servers in the cloud are live, not hot/cold standbys

what if database fails? - do we detect this, and automatically put the servers into maintenance mode?

• if not, the fear here is that the agents will thrash because they attempt to fail over to other servers that can take their load...but no servers can take that load because they are all dead in the water with no available database to accept that data

  • handle db connection failures, trap them at the comm layer and turn them into "server is no good" for the agent, so that the agents can initiated fail over (but only after they back-off, try again, and reach their fail over threshold)

• but...maybe only this one server's connection to the db has been severed, and there exists a different route that the agent can find through one of its backups

server-side control

• use clustered quartz, which ensures only one server is doing some work at some point in time

  • so only one guy in the cloud periodically repartitions the agents according to heuristic

• SuspectAgent job in HA environment: might need modification because agents aren't down if some server can't contact it, an agent should only be suspect if NO server can contact it

agent knows about one primary server, and (N-1) ordered backups (i.e., the agent knows about every server)

• info is persisted in db, in new tables that track HA metadata

fail over support

• initiated when server goes down (crash, shutdown, loss of connectivity [after exponential back-off threshold is met], entering a "hard" maintenance mode)

• agent attaches to next in its list, if available, and so on...

  • if agent gets to the end of the list, it waits for a period, and then starts again with the first server in the list, i.e. agents never just give up

• agent will not attach to servers that are not ready (next server in the list might also be sick)

maybe not fail over so quickly? - concern is thrashing

• • exponentially back-off the sending interval when the server appears to be having problems accepting load (as opposed to being down completely [which would immediately trigger a fail over to begin]) before attempting to fail over (use threshold to determine when fail over should really occur)

  • linear sliding back to a quick sending interval (no faster than basal period of 1 min) – if we starting sending again too quickly, then we might bombard a server with backlogged data from a temporary network blip.

data flow

• each server is responsible for full bidirectional communication with currently attached agents

• however, any server can talk to any agent because the clustered quartz job request can trigger on a server that isn't some agent's current primary - but that's OK because requests should be small

  • the agent will send any results back up through that agent's primary, not through the server it got the request from - we want the possibly large response to be spread across servers not centralized back through the server getting the quartz job triggers

use clustered quartz job so that only one server in the cloud executes the repartition job at a time, on a periodic basis, and according to heuristic

if a new server joins the cloud, the distribution protocol runs immediately to repartition the network to take advantage of the new resource

• by "join" we mean either a previously unknown server or a known server coming (back) online

if an agent joins / register, we examine the current topology and make the best choice for how to generate this agent's server-list, in other words we do not need to repartition the entire network

• also, it's possible that multiple agents are trying to register simultaneously - this is ok. the window for this to happen is small to begin with, but even if a non-optimal choice is made at this point (because multiple servers are generating the server-lists based off the same information, as opposed to only letting one of them do it at a time), the agents will rebalance when the quartz job run and re-examines the topology as a whole

algorithm needs to be computationally efficient (approaching O(1) time) to sustain anticipated 10K agent / 100 server setup

• algorithm can be a weighted sum of: # agents, # metrics / min, time it takes to process X metrics (relative processing #s), # cores

• # cores will be exposed in the HA admin console as relative weights for each server, by default all servers will have 1 weight / core

need to implement one (or more) thresholds so that minor changes to measurement schedules / # agents do not cause unnecessary thrashing (i.e., don't repartition unless the average difference in load is greater than some threshold value)

alerts cache

• doesn't need to be clustered, because all in-progress data (such as dampening events) are persisted

• so we can blindly have the agent data follow the distribution protocol, i.e. when a new agent connects to the server, alerts cache pulls in definitions relevant to that agent

web session repl - not in 1.1.0

• can be clustered, but not much benefit - the only things that use the session today are multi-page UI flows, nothing critical

alerts cache

• server loads data at startup by inspecting HA table and noticing which agents this server is marked as a primary for (as opposed to today where it loads all alert conditions for all alert definitions as well as creates OOBs for all measurement baselines for all measurement schedules)

• also, server loads data at agent connection time (if this is a new agent in the infrastructure or if the agent is failing over from a different server), and blocks the agent from sending data until it is finished warming up this cache data

  • loads alert conditions for alert definitions on resources that are managed by this agent

  • creates OOB conditions for measurement baselines for measurement schedules on resources that are managed by this agent

• implementation changes

  • today, we write-lock the entire cache when it loads, and write lock certain portion of the cache when it is updated

  • tomorrow, do we even need locking anymore?

    • yes, because we have a dual-loading model: some data at startup, and some at agent connection time

    • no need to change the locking model until performance becomes an issue

      • we could lock on a per-resource level, but this is probably too granular and could be overheard if we tried to keep thousands and thousands of locks for the thousands of resources that would be amnaged by our handful of agents

      • we could lock on a per-agent level, seems a healthy balance of # locks and improved performance...but, again, unnecessary until we find that locking the entire cache for the infrequent updates caused by changing the inventory hierarchy or alert definition conditions is really necessary

• cleanup

  • if an agent thinks a server has crashed when it really hasn't, the agent will fail over to another server

  • the fail over process, when the agent connects to its secondary, will update the HA tracking tables in the db

  • need a way to tell the former server that it no longer has to manage that agent (because we want specific servers to handle certain quartz jobs) / to clear the stale cache data out of a server that no longer manages that agent

    • have each server periodically check the HA table, determine whether it is really still managing this agent, and expunge stale data for agents whose primary is not itself

even though there are many servers in the cloud, an agent still chooses to register with a single server - it doesn't matter which one

server receives registration request, and computes fail-over list for that single agent only

• no need to compute all lists for all agents here, because if each server does this work each time a new agent is imported, the load can be incrementally balanced by the distribution algorithm - and if imbalanced are introduced, they will be eliminated the next time the quartz job runs to do a full network analyze and possible large-scale repartition

agent receives this fail over list and begins the "connection phase" to the 1st items in its server-list

• yes, this means that an agent might be immediately redirected to communicate with some other server in the cloud, not the one is just registered with - everything is driven now from the server-list that each agent has

• this makes setting up agents en masse much simpler because a single agent-configuration.xml file can be used across ALL agents - i.e., they all initially connect to the same server for registration - but then they get dispersed across the network to other servers for all other communication according to distribution algorithm

use server list to find proper server

agent will send a blocking request to the server to connect, and server must return a positive connection response

• this synchronous / blocking call is a natural way of allowing the agent to wait until the server has performed activities required before it is ready for this agent's data load

  • server will warning up caches

  • server will look at HA db tables, and attempt to contact the agent's previous server (under the theory that the agent-previousServer connection was lost, but that the currentServer-previousServer connect might still work

    • if the previousServer is still up, this server will send it an async message telling it to discard cached data for the agent it has now lost; otherwise, the currentServer is ready to continue other processing for this newly connecting agent

• if the server crashes during this time, that's OK - the agent will detect this (because the comm layer will report an exception on the remote endpoint), and the agent can use its backup list to fail over to its secondary, tertiary, etc

agent restart - when it starts, it connects to the primary server in its list (again, server-list drives the agent->server comm now)

identification

• servers need to be identified as being in a particular, named datacenter

• agents need to be identified as being in a particular, named datacenter

installation / registration

• when a server is installed, it needs to identify what datacenter it belongs to, otherwise it gets put into the named, unknown datacenter

• when an agent registers with some server in the cloud, it initially gets put into the datacenter that the server belongs to (this is only for ease of installation, but will be configurable via the HA admin console)

administration

• datacenter membership information needs to be represented in the HA administrative console, and RHQ admins need to be able to change this group membership information easily

distribution algorithm

• prefer to connect to servers in your own datacenter first, secondary based on load; if necessary, agents can still fail over to another datacenter – should "affinity to a particular datacenter" versus "agent load distribution" be configurable parameters so that admins with different networks can have different affinities accordingly?

if no db exist yet, install it cause i'm the first guy

if old db, upgrade it cause i'm the first guy (servers started up later will already see that the db is upgraded)

all servers need to inject themselves into the db now, so we need to extend the schema to track these

datacenter considerations

• don't require user to fill in AffinityGroup info for simple, single-server installations

  • just assume the default, unnamed DC so that all server are put into the same DC

• require user to fill in AffinityGroup information for the agent

  • if not, assume NO affinity group

if agents use their server list to reconfigure themselves, then they will attempt to do so during an upgrade procedure as well (since the servers go down for the upgrade), so we need to support a "maintenance mode", wherein agents will temporarily suspend their HA logic so as to prevent thrashing

• use our concurrency framework and set things to "zero"

• perhaps "flip the bit" on the underlying invoker framework?

develop "maintenance mode" mechanism?

• hard maintenance mode: used primarily for testing purposes, it simulates a server going down without the server actually going down (something like shutting down the low-level communication mechanism to make agents think this server has failed); this way, a toggle button within a centralized UI can be leveraged to more rapidly and more conveniently test the failover mechanism is working as intended

• soft maintenance mode: all about putting the agents not servers into a maintenance mode, so that proactive (scheduled ops, etc) and reactive (alerting, etc) elements are disabled for that agent; this is useful for when you know you will be performing maintenance on some managed server and you want to suppress non-monitoring effects you have registered against resources managed by that agent

add proto vers identifier in the message to connect

• version string must match on the server and agent side, otherwise comm is disallowed

• if any side is missing the version id (pre 1.1 instance), communication is disallowed